Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Mongo 3.6 #9632

Closed

Conversation

hwillson
Copy link
Contributor

@hwillson hwillson commented Feb 2, 2018

Hi all - this PR reverts the revert (85d74f8) of dfc0702, and includes fixes to address the previous test failures. I've labelled this as a work in progress, as I think it makes sense to also address the following issues, in this PR:

The tests will of course fail here until a new dev_bundle is generated / published, but we can hold off on that until the above items are addressed.

I'll re-cap the previous Mongo 3.6 PR notes (#9533), so we don't have to jump around:

Fixes #9623.


This PR updates the Meteor Tool to use Mongo 3.6.2 for 64-bit OS' and Mongo 3.2.18 for 32-bit OS' (I've left the garbage BUNDLE_VERSION in place intentionally, since these changes will require a new dev_bundle publish from MDG).

A few important mentions:

  • As of Mongo 3.6, all Mongo binary downloads include SSL - there is no longer a non-SSL based download bundle (so it's a bit bigger, but that shouldn't be an issue).
  • Using the --nojournal option with WiredTiger based replica sets is no longer supported (see https://jira.mongodb.org/browse/SERVER-30760). The --nojournal flag was added in bcfe072 to help reduce the amount of disk space used by Mongo, but since this option is no longer supported, we'll have to live with the extra disk space usage.

Mongo Node driver side note:

The Mongo Node driver has recently jumped into the 3.x range, and has introduced a few breaking changes. While we normally bump the Mongo Node driver to the most recent version when doing Mongo updates, I've left it in 2.x range for now. I'll submit a separate Mongo Node driver 3.x update PR, so we can keep the focus on the Mongo upgrade here. Mongo 3.6 will work with 2.x versions of the Mongo Node driver.

Thanks!

@abernix
Copy link
Contributor

abernix commented Feb 6, 2018

It may be worth clarifying that, while Windows and Mac do only come bundled with SSL, SSL is still not included with the Generic Linux tarball (which scripts/generate-dev-bundle.sh pulls in), presumably since cross-distro linking of SSL libraries is more difficult. I snuck this change which accommodates that rule into the last PR, in order to build the dev bundle: fce09ac#diff-be3b40b6bc1b45a60ad8698bc2065cb9R57.

I've built the dev bundle, though we'll want to remember to quell e73013b before merging (as notated with the [DELETEME] prefix to the commit message).

Thanks for looking at this, @hwillson!

@hwillson
Copy link
Contributor Author

The last part of this PR is to figure out what to do about #9591. When meteor update'ing to a version of the Tool that uses Mongo 3.6, jumping back to run an older version of Meteor (using meteor --release X) can break. This happens as the Mongo 3.6 updated database is no longer directly usable by Mongo 3.4 (or earlier). Why is the database no longer directly usable? Good question - I haven't been able to track the reason down (nothing in the logs, everything looks okay with the database, etc.). This might be caused by the fact that the Mongo 3.6 database is updated to use journaling by default, then we're jumping back to a setup that expects no journaling, but my tests in this area don't point to this being a problem.

The database can be repaired (see #9591 (comment)) and used with previous versions of Meteor, but this is a manual step. We can't really automate this as we'd then have to release new patch versions for older Meteor releases, to include the code to handle the database repair when the Mongo startup error happens. Due to the way Tool springboarding works, we can't just add code to handle the repair in Meteor 1.6.2, and expect to have it called when running meteor --release X. Given this, the best option here might just be to document this potential issue, as well as the mongod --repair fix, and make sure people are aware of the potential issue.

@hwillson
Copy link
Contributor Author

Just re-capping the decision from our call - I'll document the potential issue mentioned in #9632 (comment), along with the Mongo repair fix, in the History.md. We should then be all set to get this merged. Updates coming shortly - thanks!

@hwillson
Copy link
Contributor Author

hwillson commented Mar 1, 2018

This should now be ready for a new dev_bundle build/deploy (removing WIP). Thanks!

@hwillson hwillson changed the title [WIP] Upgrade to Mongo 3.6 Upgrade to Mongo 3.6 Mar 1, 2018
@benjamn benjamn self-assigned this Mar 7, 2018
@benjamn benjamn added this to the Release 1.6.2 milestone Mar 7, 2018
@abernix abernix self-assigned this Mar 12, 2018
@abernix abernix force-pushed the mongo-3.6-upgrade-with-test-fix branch 2 times, most recently from d264cc2 to c9a31a1 Compare March 13, 2018 08:39
@abernix
Copy link
Contributor

abernix commented Mar 13, 2018

@hwillson I've re-published dev bundle 9999.9632.1 and the non-Windows tests passed, but I've just fixed the "merge conflict" so that AppVeyor will run.

@abernix abernix force-pushed the mongo-3.6-upgrade-with-test-fix branch from b4784ca to e69754f Compare March 21, 2018 16:08
The accounts system `logoutOtherClients` method is using
a connection close delay to make sure clients have a chance
to get updated tokens, before older tokens are removed.
Certain tests are setting this delay to 0, to help speed
things up. The Meteor Mongo 3.6 changes have brought back in
the need to use Mongo journaling, since the ability to disable
journaling has been deprecated by Mongo. This means there is
now a slight performance hit on certain Mongo operations,
due to the increased time needed to cover the write to disk.
This slight performance hit is causing the tests that use the
0 connection close delay to fail, since they are not waiting
long enough for clients to get updated tokens, before they're
removed. These changes increase the test connection close
delay, to fix the failing tests.
@abernix abernix force-pushed the mongo-3.6-upgrade-with-test-fix branch from e69754f to 84e0d94 Compare March 21, 2018 17:27
@abernix
Copy link
Contributor

abernix commented Mar 21, 2018

Merged to devel as 3903020.

@abernix abernix closed this Mar 21, 2018
@bjarketrux
Copy link

Mongo 3.6 will work with 2.x versions of the Mongo Node driver.

Note: According to this page Node.js Driver Compatibility MongoDB 3.6 is only compatible with Node.js Driver >= 3.0.

I havn't investigated further.

@glasser
Copy link
Contributor

glasser commented May 22, 2018

@bjarketrux I believe that this means that not all features of MongoDB 3.6 are supported by the 2.x driver, but it doesn't like completely break.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New $v field in Mongo 3.6 oplog breaks OplogObserveDriver
5 participants